fdir

What is fdir?

The fdir npm package is a fast and efficient directory crawler that allows you to read and list files in a directory tree. It is designed to be highly performant and can handle large directories with ease.

What are fdir's main functionalities?

Basic Directory Crawling

This feature allows you to crawl a directory and get a list of all files with their full paths. The code sample demonstrates how to use fdir to crawl a directory and log the list of files.

const { fdir } = require('fdir');
const api = new fdir().withFullPaths().crawl('path/to/directory');
api.withPromise().then(files => console.log(files));

Filtering Files

This feature allows you to filter the files based on a condition. The code sample shows how to filter files to include only JavaScript files.

const { fdir } = require('fdir');
const api = new fdir().withFullPaths().filter(path => path.endsWith('.js')).crawl('path/to/directory');
api.withPromise().then(files => console.log(files));

Using Streams

This feature allows you to use streams to handle the files as they are found. The code sample demonstrates how to use a stream to log each file as it is crawled.

const { fdir } = require('fdir');
const api = new fdir().withFullPaths().crawl('path/to/directory');
api.withStream().on('data', file => console.log(file));

Using Async/Await

This feature allows you to use async/await syntax for better readability and control over asynchronous operations. The code sample shows how to use async/await to crawl a directory and log the list of files.

const { fdir } = require('fdir');
(async () => {
  const api = new fdir().withFullPaths().crawl('path/to/directory');
  const files = await api.withPromise();
  console.log(files);
})();

Other packages similar to fdir

fast-glob

fast-glob is a library for searching files by glob patterns. It is highly performant and supports advanced features like negative globs and multiple patterns. Compared to fdir, fast-glob is more focused on pattern matching and offers a rich set of options for globbing.

readdirp

readdirp is a recursive version of fs.readdir with additional features like filtering and streaming. It is similar to fdir in that it allows you to read directories recursively, but it is less focused on performance and more on providing a simple API for directory traversal.

klaw

klaw is a file system walker that uses streams to traverse directories. It is similar to fdir in its use of streams but is generally slower. klaw is more suitable for applications where streaming is a primary requirement.

README

The Fastest Directory Crawler & Globber for NodeJS

⚡ The Fastest: Nothing similar (in the NodeJS world) beats fdir in speed. It can easily crawl a directory containing 1 million files in < 1 second.

💡 Stupidly Easy: fdir uses expressive Builder pattern to build the crawler increasing code readability.

🤖 Zero Dependencies*: fdir only uses NodeJS fs & path modules.

🕺 Astonishingly Small: < 2KB in size gzipped & minified.

🔥 All Node Versions Supported: Unlike other similar libraries that have dropped support for Node versions < 10, fdir supports all versions >= 6.

🖮 Hackable: Extending fdir is extremely simple now that the new Builder API is here. Feel free to experiment around.

* picomatch must be installed manually by the user to support globbing.

Support

Do you like this project? Support me by donating, creating an issue, becoming a stargazer, or opening a pull request. Thanks.

Status

This project is not dead. In terms of performance, I don't think there is anything more to be done. This is as fast a NodeJS Directory Crawler as you can get. I could be wrong though.

What's next:

1. There is still no Async Iterator API (contributions welcome).
2. We need to discuss whether its feasible (or even sensible) to have event emitters in different parts of the crawling process.
3. Make globbing pluggable
4. Finalize and freeze the API
5. Improve documentation (maybe we can make a proper documentation website?)

Contributions are welcome in all these.

🚄 Quickstart

Installation

You can install using npm:

$ npm i fdir

or Yarn:

$ yarn add fdir

Usage

const { fdir } = require("fdir");

// create the builder
const api = new fdir().withFullPaths().crawl("path/to/dir");

// get all files in a directory synchronously
const files = api.sync();

// or asynchronously
api.withPromise().then((files) => {
  // do something with the result here.
});

Documentation:

I have written in-depth documentation here.

📊 Benchmarks:

Specs:

• CPU: Intel i7 7th Generation (7700HQ)
• RAM: 16 GB
• Storage: 256 GB SSD
• OS: Manjaro Linux
• Directory Size: < 2k files

Notes:

• Some people asked that I benchmark no-op (without options) version of fdir.
  • I did and found no performance difference. The results were identical. (I didn't include it here as it wasn't anything special.)
• Some other people were doubtful about the authenticity of these results due to frequency scaling, process overload, disk warmup, etc.
  • So I have updated the benchmark with new results that should resolve all those doubts. Here's the process I followed:
  • Hard shutdown the laptop (a couple of times just to be sure) to clear disk, ram cache etc.
  • Login directly to a TTY (avoiding any unnecessary process from starting).
  • Disable CPU Scaling using

    $ sudo cpupower frequency-set --governor performance
    

  • Run the benchmark

The Fastest Globber

Last updated: May 13, 2020 (fdir v3.3.0)

$ yarn bench:glob

glob pattern used: **.js & **/**.js

Node v13.13.0

Synchronous	Asynchronous

The Fastest Directory Crawler

Last updated: May 10, 2020 (fdir v3.0.0)

$ yarn bench

Node v14.2.0:

Synchronous	Asynchronous

Node v8.7.0:

Older versions of fdir (1.x & 2.x) used synchronous lstat call (lstatSync) in the asynchronous API to acheive speed on Node < 10. This has been fixed in fdir 3.0.0.

Synchronous

🦮 LICENSE

Copyright © 2020 Abdullah Atta under MIT. Read full text here.